Momentum contrast transformer for COVID-19 diagnosis with knowledge distillation.

Intelligent diagnosis has been widely studied in diagnosing novel corona virus disease (COVID-19). Existing deep models typically do not make full use of the global features such as large areas of ground glass opacities, and the local features such as local bronchiolectasis from the COVID-19 chest CT images, leading to unsatisfying recognition accuracy. To address this challenge, this paper proposes a novel method to diagnose COVID-19 using momentum contrast and knowledge distillation, termed MCT-KD. Our method takes advantage of Vision Transformer to design a momentum contrastive learning task to effectively extract global features from COVID-19 chest CT images. Moreover, in transfer and fine-tuning process, we integrate the locality of convolution into Vision Transformer via special knowledge distillation. These strategies enable the final Vision Transformer simultaneously focuses on global and local features from COVID-19 chest CT images. In addition, momentum contrastive learning is self-supervised learning, solving the problem that Vision Transformer is challenging to train on small datasets. Extensive experiments confirm the effectiveness of the proposed MCT-KD. In particular, our MCT-KD is able to achieve 87.43% and 96.94% accuracy on two publicly available datasets, respectively.

FedUSC: Collaborative Unsupervised Representation Learning from Decentralized Data for Internet of Things

Federated Learning (FL) lately has shown much promise in improving the shared model and preserving data privacy. However, these existing methods are only of limited utility in the Internet of Things (IoT) scenarios, as they either heavily depend on high-quality labeled data or only perform well under idealized conditions, which typically cannot be found in practical applications. In this paper, we propose a novel federated unsupervised learning method for image classification without the use of any ground truth annotations. In IoT scenarios, a big challenge is that decentralized data among multiple clients is normally non-IID, leading to performance degradation. To address this issue, we further propose a dynamic update mechanism that can decide how to update the local model based on weights divergence. Extensive experiments show that our method outperforms all baseline methods by large margins, including +6.67% on CIFAR-10, +5.15% on STL-10, and +8.44% on SVHN in terms of classification accuracy. In particular, we obtain promising results on Mini-ImageNet and COVID-19 datasets and outperform several federated unsupervised learning methods under non-IID settings. IEEE

Fusing hybrid attentive network with self-supervised dual-channel heterogeneous graph for knowledge tracing

Recently the large-scale influence of Covid-19 promoted the fast development of intelligent tutoring systems (ITS). As a major task of ITS, Knowledge Tracing (KT) aims to capture a student's dynamic knowledge state based on his historical response sequences and provide personalized learning assistance to him. However, most existing KT methods have encountered the data sparsity problem. In real scenarios, an online tutoring system usually has an extensive collection of questions while each student can only interact with a limited number of questions. As a result, the records of some questions could be extremely sparse, which degrades the performance of traditional KT models. To resolve this issue, we propose a Dual-channel Heterogeneous Graph Network (DHGN) to learn informative representations of questions from students' records by capturing both the high-order heterogeneous and local relations. As the supervised learning manner applied in previous methods is incapable of exploiting unobserved relations between questions, we innovatively integrate a self-supervised framework into the KT task and employ contrastive learning via the two channels of DHGN, supplementing as an auxiliary task to improve the KT performance. Moreover, we adopt the attention mechanism, which has achieved impressive performance in natural language processing tasks, to effectively capture students' knowledge state. But the standard attention network is inapplicable to the KT task because the current knowledge state of a student usually shows strong dependency on his recently interactive questions, unlike the situation of language processing tasks, which focus more on the long-term dependency. To avoid the inefficiency of standard attention networks in the KT task, we further devise a novel Hybrid Attentive Network (HAN), which produces both the global attention and the hierarchical local attention to model the long-term and short-term intents, respectively. Then, by the gating network, a student's long-term and short-term intents are combined for efficient prediction. We conduct extensive experiments on several real-world datasets. Experimental results demonstrate that our proposed methods achieve significant performance improvement compared to existing state-of-the-art baselines, which validates the effectiveness of the proposed dual-channel heterogeneous graph framework and hybrid attentive network. © 2023 Elsevier Ltd

A Deep Learning Powered System to Lie Detection While Online Study

Many education facilities have recently switched to online learning due to the COVID-19 pandemic. The nature of online learning makes it easier for dishonest behaviors, such as cheating or lying during lessons. We propose a new artificial intelligence - powered solution to help educators solve this rising problem for a fairer learning environment. We created a visual representation contrastive learning method with the MobileNetV2 network as the backbone to improve predictability from an unlabeled dataset which can be deployed on low power consumption devices. The experiment shows an accuracy of up to 59%, better than several previous research, proving the usability of this approach.

CMC_v2: Towards More Accurate COVID-19 Detection with Discriminative Video Priors

This paper presents our solution for the 2nd COVID-19 Competition, occurring in the framework of the AIMIA Workshop at the European Conference on Computer Vision (ECCV 2022). In our approach, we employ the winning solution last year which uses a strong 3D Contrastive Mixup Classification network (CMC_v1) as the baseline method, composed of contrastive representation learning and mixup classification. In this paper, we propose CMC_v2 by introducing natural video priors to COVID-19 diagnosis. Specifically, we adapt a pre-trained (on video dataset) video transformer backbone to COVID-19 detection. Moreover, advanced training strategies, including hybrid mixup and cutmix, slice-level augmentation, and small resolution training are also utilized to boost the robustness and the generalization ability of the model. Among 14 participating teams, CMC_v2 ranked 1st in the 2nd COVID-19 Competition with an average Macro F1 Score of 89.11%. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

COVID-19 diagnosis utilizing wavelet-based contrastive learning with chest CT images.

The pandemic caused by the coronavirus disease 2019 (COVID-19) has continuously wreaked havoc on human health. Computer-aided diagnosis (CAD) system based on chest computed tomography (CT) has been a hotspot option for COVID-19 diagnosis. However, due to the high cost of data annotation in the medical field, it happens that the number of unannotated data is much larger than the annotated data. Meanwhile, having a highly accurate CAD system always requires a large amount of labeled data training. To solve this problem while meeting the needs, this paper presents an automated and accurate COVID-19 diagnosis system using few labeled CT images. The overall framework of this system is based on the self-supervised contrastive learning (SSCL). Based on the framework, our enhancement of our system can be summarized as follows. 1) We integrated a two-dimensional discrete wavelet transform with contrastive learning to fully use all the features from the images. 2) We use the recently proposed COVID-Net as the encoder, with a redesign to target the specificity of the task and learning efficiency. 3) A new pretraining strategy based on contrastive learning is applied for broader generalization ability. 4) An additional auxiliary task is exerted to promote performance during classification. The final experimental result of our system attained 93.55%, 91.59%, 96.92% and 94.18% for accuracy, recall, precision, and F1-score respectively. By comparing results with the existing schemes, we demonstrate the performance enhancement and superiority of our proposed system.

Curriculum Contrastive Learning for COVID-19 FAQ Retrieval

Medical Frequently Asked Question (FAQ) retrieval aims to find the most relevant question-answer pairs for a given user query, which is of great significance for enhancing people medical health awareness and knowledge. However, due to medical data privacy and labor-intensive labeling, there is a lack of large-scale question-matching training datasets. Previous methods directly use the collected question-answer pairs on search engines to train retrieval models, which achieved poor performance. Inspired by recent advances in contrastive learning, we propose a novel contrastive curriculum learning framework for modeling user medical queries. First, we design different data augmentation methods to generate positive samples and different types of negative samples. Second, we propose a curriculum learning strategy that associates difficulty levels with negative samples. Through a contrastive learning process from easy to hard, our method achieves excellent results on two medical datasets. © 2022 IEEE.

Contrastive domain adaptation with consistency match for automated pneumonia diagnosis.

Pneumonia can be difficult to diagnose since its symptoms are too variable, and the radiographic signs are often very similar to those seen in other illnesses such as a cold or influenza. Deep neural networks have shown promising performance in automated pneumonia diagnosis using chest X-ray radiography, allowing mass screening and early intervention to reduce the severe cases and death toll. However, they usually require many well-labelled chest X-ray images for training to achieve high diagnostic accuracy. To reduce the need for training data and annotation resources, we propose a novel method called Contrastive Domain Adaptation with Consistency Match (CDACM). It transfers the knowledge from different but relevant datasets to the unlabelled small-size target dataset and improves the semantic quality of the learnt representations. Specifically, we design a conditional domain adversarial network to exploit discriminative information conveyed in the predictions to mitigate the domain gap between the source and target datasets. Furthermore, due to the small scale of the target dataset, we construct a feature cloud for each target sample and leverage contrastive learning to extract more discriminative features. Lastly, we propose adaptive feature cloud expansion to push the decision boundary to a low-density area. Unlike most existing transfer learning methods that aim only to mitigate the domain gap, our method instead simultaneously considers the domain gap and the data deficiency problem of the target dataset. The conditional domain adaptation and the feature cloud generation of our method are learning jointly to extract discriminative features in an end-to-end manner. Besides, the adaptive feature cloud expansion improves the model's generalisation ability in the target domain. Extensive experiments on pneumonia and COVID-19 diagnosis tasks demonstrate that our method outperforms several state-of-the-art unsupervised domain adaptation approaches, which verifies the effectiveness of CDACM for automated pneumonia diagnosis using chest X-ray imaging.

Curriculum Contrastive Learning for COVID-19 FAQ Retrieval

Medical Frequently Asked Question (FAQ) retrieval aims to find the most relevant question-answer pairs for a given user query, which is of great significance for enhancing people medical health awareness and knowledge. However, due to medical data privacy and labor-intensive labeling, there is a lack of large-scale question-matching training datasets. Previous methods directly use the collected question-answer pairs on search engines to train retrieval models, which achieved poor performance. Inspired by recent advances in contrastive learning, we propose a novel contrastive curriculum learning framework for modeling user medical queries. First, we design different data augmentation methods to generate positive samples and different types of negative samples. Second, we propose a curriculum learning strategy that associates difficulty levels with negative samples. Through a contrastive learning process from easy to hard, our method achieves excellent results on two medical datasets. © 2022 IEEE.

Construction and Analysis of Surrounding Travel Demanding Graph Based on Dual Contrastive Learning Text Classification and Graph Neural Network

Understanding the main information about the current situation of the tourism market has become an urgent need and new trends in the development of the tourism market. In this paper, we use natural language processing technology to analyze the development of tourism around Maoming City, Guangdong Province during the COVID-19 epidemic by means of data mining methods to build a local tourism graph, refine and design models and methods such as RoBERTa-BiGRU-Attention fusion model, dual contrastive learning, BERT-BiLSTM-CRF named entity identification technique, improved Apriori algorithm, GNNLP model based on conventional models and proved the rationality and efficiency of the improved model by comparative test, provide oriented suggestions to help government departments promote tourism and tourism enterprises product supply, optimize resource allocation and explore the market constantly during the epidemic period after scientific analysis and summary. © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)

Self-supervised heterogeneous hypergraph network for knowledge tracing

Recently online intelligent education has caught more and more attention, especially due to the global influence of Covid-19. A major task of intelligent education is Knowledge Tracing (KT) which aims to capture students' dynamic status based on their historical interaction records and predict their responses to new questions. However, most existing KT methods suffer from the record data sparsity problem. In reality, there are a huge number of questions in the online database and students can only interact with a very small set of these questions. The records of some questions could be extremely sparse, which may significantly degrade the performance of traditional KT methods. Although recent graph neural network (GNN) based KT methods can fuse graph-structured information and improve the representation of questions to some extent, the pairwise structure of GNN neglects the complex high-order and heterogeneous relations among questions. To resolve the above issues, we develop a novel KT model with the heterogeneous hypergraph network (HHN) and propose an attentive mechanism, including intra- and inter-graph attentions, to aggregate neighbors' information upon HHN. To further enhance the question representation, we supplement the supervised prediction task of KT with an auxiliary self-supervised task, i.e., we additionally generate an augmented view with adaptive data augmentation to implement contrastive learning and exploit the unobserved relations among questions. We conduct extensive experiments on several real-world datasets. Experimental results demonstrate that our proposed method achieves significant performance improvement compared to some state-of-the-art KT methods.

A self-supervised COVID-19 CT recognition system with multiple regularizations.

The diagnosis of Coronavirus Disease 2019 (COVID-19) exploiting machine learning algorithms based on chest computed tomography (CT) images has become an important technology. Though many excellent computer-aided methods leveraging CT images have been designed, they do not possess sufficiently high recognition accuracy. Besides, these methods entail vast amounts of training data, which might be difficult to be satisfied in some real-world applications. To address these two issues, this paper proposes a novel COVID-19 recognition system based on CT images, which has high recognition accuracy, while only requiring a small amount of training data. Specifically, the system possesses the following three improvements: 1) Data: a novel redesigned BCELoss that incorporates Label Smoothing, Focal Loss, and Label Weighting Regularization (LSFLLW-R) technique for optimizing the solution space and preventing overfitting, 2) Model: a backbone network processed by two-phase contrastive self-supervised learning for classifying multiple labels, and 3) Method: a decision-fusing ensemble learning method for getting a more stable system, with balanced metric values. Our proposed system is evaluated on the small-scale expanded COVID-CT dataset, achieving an accuracy of 94.3%, a precision of 94.1%, a recall (sensitivity) of 93.4%, an F1-score of 94.7%, and an Area Under the Curve (AUC) of 98.9%, for COVID-19 diagnosis, respectively. These experimental results verify that our system can not only identify pathological locations effectively, but also achieve better performance in terms of accuracy, generalizability, and stability, compared with several other state-of-the-art COVID-19 diagnosis methods.

Contrastive learning and subtyping of post-COVID-19 lung computed tomography images.

Patients who recovered from the novel coronavirus disease 2019 (COVID-19) may experience a range of long-term symptoms. Since the lung is the most common site of the infection, pulmonary sequelae may present persistently in COVID-19 survivors. To better understand the symptoms associated with impaired lung function in patients with post-COVID-19, we aimed to build a deep learning model which conducts two tasks: to differentiate post-COVID-19 from healthy subjects and to identify post-COVID-19 subtypes, based on the latent representations of lung computed tomography (CT) scans. CT scans of 140 post-COVID-19 subjects and 105 healthy controls were analyzed. A novel contrastive learning model was developed by introducing a lung volume transform to learn latent features of disease phenotypes from CT scans at inspiration and expiration of the same subjects. The model achieved 90% accuracy for the differentiation of the post-COVID-19 subjects from the healthy controls. Two clusters (C1 and C2) with distinct characteristics were identified among the post-COVID-19 subjects. C1 exhibited increased air-trapping caused by small airways disease (4.10%, p = 0.008) and diffusing capacity for carbon monoxide %predicted (DLCO %predicted, 101.95%, p < 0.001), while C2 had decreased lung volume (4.40L, p < 0.001) and increased ground glass opacity (GGO%, 15.85%, p < 0.001). The contrastive learning model is able to capture the latent features of two post-COVID-19 subtypes characterized by air-trapping due to small airways disease and airway-associated interstitial fibrotic-like patterns, respectively. The discovery of post-COVID-19 subtypes suggests the need for different managements and treatments of long-term sequelae of patients with post-COVID-19.

Unsupervised Contrastive Learning of Image Representations from Ultrasound Videos with Hard Negative Mining

Rich temporal information and variations in viewpoints make video data an attractive choice for learning image representations using unsupervised contrastive learning (UCL) techniques. State-of-the-art (SOTA) contrastive learning techniques consider frames within a video as positives in the embedding space, whereas the frames from other videos are considered negatives. We observe that unlike multiple views of an object in natural scene videos, an Ultrasound (US) video captures different 2D slices of an organ. Hence, there is almost no similarity between the temporally distant frames of even the same US video. In this paper we propose to instead utilize such frames as hard negatives. We advocate mining both intra-video and cross-video negatives in a hardness-sensitive negative mining curriculum in a UCL framework to learn rich image representations. We deploy our framework to learn the representations of Gallbladder (GB) malignancy from US videos. We also construct the first large-scale US video dataset containing 64 videos and 15,800 frames for learning GB representations. We show that the standard ResNet50 backbone trained with our framework improves the accuracy of models pretrained with SOTA UCL techniques as well as supervised pretrained models on ImageNet for the GB malignancy detection task by 2–6%. We further validate the generalizability of our method on a publicly available lung US image dataset of COVID-19 pathologies and show an improvement of 1.5% compared to SOTA. Source code, dataset, and models are available at https://gbc-iitd.github.io/usucl. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Self-supervised contrastive learning for integrative single cell RNA-seq data analysis.

We present a novel self-supervised Contrastive LEArning framework for single-cell ribonucleic acid (RNA)-sequencing (CLEAR) data representation and the downstream analysis. Compared with current methods, CLEAR overcomes the heterogeneity of the experimental data with a specifically designed representation learning task and thus can handle batch effects and dropout events simultaneously. It achieves superior performance on a broad range of fundamental tasks, including clustering, visualization, dropout correction, batch effect removal, and pseudo-time inference. The proposed method successfully identifies and illustrates inflammatory-related mechanisms in a COVID-19 disease study with 43 695 single cells from peripheral blood mononuclear cells.

Self-supervised region-aware segmentation of COVID-19 CT images using 3D GAN and contrastive learning.

Medical image segmentation is a key initial step in several therapeutic applications. While most of the automatic segmentation models are supervised, which require a well-annotated paired dataset, we introduce a novel annotation-free pipeline to perform segmentation of COVID-19 CT images. Our pipeline consists of three main subtasks: automatically generating a 3D pseudo-mask in self-supervised mode using a generative adversarial network (GAN), leveraging the quality of the pseudo-mask, and building a multi-objective segmentation model to predict lesions. Our proposed 3D GAN architecture removes infected regions from COVID-19 images and generates synthesized healthy images while keeping the 3D structure of the lung the same. Then, a 3D pseudo-mask is generated by subtracting the synthesized healthy images from the original COVID-19 CT images. We enhanced pseudo-masks using a contrastive learning approach to build a region-aware segmentation model to focus more on the infected area. The final segmentation model can be used to predict lesions in COVID-19 CT images without any manual annotation at the pixel level. We show that our approach outperforms the existing state-of-the-art unsupervised and weakly-supervised segmentation techniques on three datasets by a reasonable margin. Specifically, our method improves the segmentation results for the CT images with low infection by increasing sensitivity by 20% and the dice score up to 4%. The proposed pipeline overcomes some of the major limitations of existing unsupervised segmentation approaches and opens up a novel horizon for different applications of medical image segmentation.

A Deep Learning Powered System to Lie Detection While Online Study

Many education facilities have recently switched to online learning due to the COVID-19 pandemic. The nature of online learning makes it easier for dishonest behaviors, such as cheating or lying during lessons. We propose a new artificial intelligence - powered solution to help educators solve this rising problem for a fairer learning environment. We created a visual representation contrastive learning method with the MobileNetV2 network as the backbone to improve predictability from an unlabeled dataset which can be deployed on low power consumption devices. The experiment shows an accuracy of up to 59%, better than several previous research, proving the usability of this approach. © 2022 Lavoisier. All rights reserved.

A Contrastive Graph Convolutional Network for Toe-Tapping Assessment in Parkinson's Disease

One of the common motor symptoms of Parkinson’s disease (PD) is bradykinesia. Automated bradykinesia assessment is critically needed for helping neurologists achieve objective clinical diagnosis and hence provide timely and appropriate medical services. This need has become especially urgent after the outbreak of the coronavirus pandemic in late 2019. Currently, the main factor limiting the accurate assessment is the difficulty of mining the fine-grained discriminative motion features. Therefore, we propose a novel contrastive graph convolutional network for automated and objective toe-tapping assessment, which is one of the most important tests of lower-extremity bradykinesia. Specifically, based on joint sequences extracted from videos, a supervised contrastive learning strategy was followed to cluster together the features of each class, thereby enhancing the specificity of the learnt class-specific features. Subsequently, a multi-stream joint sparse learning mechanism was designed to eliminate potentially similar redundant features of joint position and motion, hence strengthening the discriminability of features extracted from different streams. Finally, a spatial-temporal interaction graph convolutional module was developed to explicitly model remote dependencies across time and space, and hence boost the mining of fine-grained motion features. Comprehensive experimental results demonstrate that this method achieved remarkable classification performance on a clinical video dataset, with an accuracy of 70.04% and an acceptable accuracy of 98.70%. These results obviously outperformed other existing sensor- and video-based methods. The proposed video-based scheme provides a reliable and objective tool for automated quantitative toe-tapping assessment, and is expected to be a viable method for remote medical assessment and diagnosis. IEEE

Supervised Pretraining through Contrastive Categorical Positive Samplings to Improve COVID-19 Mortality Prediction.

Clinical EHR data is naturally heterogeneous, where it contains abundant sub-phenotype. Such diversity creates challenges for outcome prediction using a machine learning model since it leads to high intra-class variance. To address this issue, we propose a supervised pre-training model with a unique embedded k-nearest-neighbor positive sampling strategy. We demonstrate the enhanced performance value of this framework theoretically and show that it yields highly competitive experimental results in predicting patient mortality in real-world COVID-19 EHR data with a total of over 7,000 patients admitted to a large, urban health system. Our method achieves a better AUROC prediction score of 0.872, which outperforms the alternative pre-training models and traditional machine learning methods. Additionally, our method performs much better when the training data size is small (345 training instances).

DCML: Deep contrastive mutual learning for COVID-19 recognition.

COVID-19 is a form of disease triggered by a new strain of coronavirus. Automatic COVID-19 recognition using computer-aided methods is beneficial for speeding up diagnosis efficiency. Current researches usually focus on a deeper or wider neural network for COVID-19 recognition. And the implicit contrastive relationship between different samples has not been fully explored. To address these problems, we propose a novel model, called deep contrastive mutual learning (DCML), to diagnose COVID-19 more effectively. A multi-way data augmentation strategy based on Fast AutoAugment (FAA) was employed to enrich the original training dataset, which helps reduce the risk of overfitting. Then, we incorporated the popular contrastive learning idea into the conventional deep mutual learning (DML) framework to mine the relationship between diverse samples and created more discriminative image features through a new adaptive model fusion method. Experimental results on three public datasets demonstrate that the DCML model outperforms other state-of-the-art baselines. More importantly, DCML is easier to reproduce and relatively efficient, strengthening its high practicality.